Papers with vanilla pre-training
STEP: Staged Parameter-Efficient Pre-training for Large Language Models (2025.naacl-short)
Copied to clipboard
| Challenge: | Recent LLM development trends involve pre-training models with a vast number of parameters on massive datasets. |
| Approach: | They propose a method that integrates parameter-efficient tuning techniques with model growth to reduce memory requirements while maintaining equivalent performance. |
| Outcome: | The proposed method reduces memory requirements by 53.9% while maintaining equivalent performance to vanilla pre-trained models on downstream tasks. |
STEP: Staged Parameter-Efficient Pre-training for Large Language Models (2024.acl-srw)
Copied to clipboard
| Challenge: | Existing methods for reducing computational costs during pre-training have been studied, but they often degrade performance under fair conditions. |
| Approach: | They propose a method that combines parameter-efficient tuning and staged training to reduce memory requirements while maintaining comparable performance. |
| Outcome: | The proposed method reduces memory requirements by 40.4% while maintaining comparable performance. |
Comprehensive Study of Bilingual and Multi-category Instruction Pre-training (2026.findings-eacl)
Copied to clipboard
| Challenge: | Instruction pre-training (IPT) has recently gained attention as an intermediate stage between pre- and post-training for large language models. |
| Approach: | They study the optimal balance between raw and instruction-response data, languages, and task categories in an LLM instruction-respondence dataset. |
| Outcome: | The proposed model improves on English-centric and bilingual models using bilingual instruction-response datasets. |